Merak: An Efficient Distributed DNN Training Framework With Automated 3D Parallelism for Giant Foundation Models

PipeDream: Pipeline Parallelism for DNN Training

An Efficient Curvelet Framework for Denoising Images

Wiener filter suppresses noise efficiently. However, it makes the out image blurred. Curvelet preserves the edges of natural images perfectly, but, it produces visual distortion artifacts and fuzzy edges to the restored image, especially in homogeneous regions of images. In this paper, a new image denoising framework based on Curvelet transform and wiener filter is proposed, which can stop nois...

An MPI-Based Python Framework for Distributed Training with Keras

We present a lightweight Python framework for distributed training of neural networks on multiple GPUs or CPUs. The framework is built on the popular Keras machine learning library. The Message Passing Interface (MPI) protocol is used to coordinate the training process, and the system is well suited for job submission at supercomputing sites. We detail the software’s features, describe its use,...

DNN-Train: Benchmarking and Analyzing DNN Training

We aim to build a new benchmark pool for deep neural network training and to analyze how eicient existing frameworks are in performing this training. We will provide our methodology and develop proper proiling tools to perform this analysis.

Scalable distributed DNN training using commodity GPU cloud computing

We introduce a new method for scaling up distributed Stochastic Gradient Descent (SGD) training of Deep Neural Networks (DNN). The method solves the well-known communication bottleneck problem that arises for data-parallel SGD because compute nodes frequently need to synchronize a replica of the model. We solve it by purposefully controlling the rate of weight-update per individual weight, whic...